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Intellectual Property Rights 



IPRs essential or potentially essential to the present document may have been declared to ETSI. The information 
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found 
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in 
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web 
server ( http://webapp.etsi.org/IPR/home.asp ). 

Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee 
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web 
server) which are, or may be, or may become, essential to the present document. 



Foreword 

This Technical Specification (TS) has been produced by ETSI Technical Committee Speech Processing, Transmission 
and Quality Aspects (STQ). 

The present document is part 5 of a multi-part deliverable covering the QoS aspects for popular services in GSM and 
3G networks, as identified below: 

Part 1 : "Identification of Quality of Service aspects" ; 

Part 2: "Definition of Quality of Service parameters and their computation"; 

Part 3: "Typical procedures for Quality of Service measurement equipment"; 

Part 4: "Requirements for Quality of Service measurement equipment"; 

Part 5: "Definition of typical measurement profiles"; 

Part 6: "Post processing and statistical methods". 

Part 7: "Sampling methodology". 

Part 1 identifies QoS aspects for popular services in GSM and 3G networks. For each service chosen QoS indicators are 
listed. They are considered to be suitable for the quantitatively characterization of the dominant technical QoS aspects 
as experienced from the end-customer perspective. 

Part 2 defines QoS parameters and their computation for popular services in GSM and 3G networks. The technical QoS 
indicators, listed in part 1, are the basis for the parameter set chosen. The parameter definition is split into two parts: the 
abstract definition and the generic description of the measurement method with the respective trigger points. Only 
measurement methods not dependent on any infrastructure provided are described in the present document. The 
harmonized definitions given in the present document are considered as the prerequisites for comparison of QoS 
measurements and measurement results. 

Part 3 describes typical procedures used for QoS measurements over GSM, along with settings and parameters for such 
measurements. 

Part 4 defines the minimum requirements of QoS measurement equipment for GSM and 3G networks in the way that 
the values and trigger-points needed to compute the QoS parameter as defined in part 2 can be measured following the 
procedures defined in part 3. Test-equipment fulfilling the specified minimum requirements, will allow to perform the 
proposed measurements in a reliable and reproducible way. 

Part 5 specifies test profiles which are required to enable benchmarking of different GSM or 3G networks both within 
and outside national boundaries. It is necessary to have these profiles so that when a specific set of tests are carried out 
then customers are comparing "like for like" performance. 
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Part 6 describes procedures to be used for statistical calculations in the field of QoS measurement of GSM and 3G 
networks using probing systems. 

Part 7 describes the field measurement method procedures used for QoS measurements over GSM where the results are 
obtained applying inferential statistics. 



Introduction 

All the defined quality of service parameters and their computations are based on field measurements. That indicates 
that the measurements were made from customers point of view (full End-to-End perspective, taking into account the 
needs of testing). 

It is assumed that the end customer can handle his mobile and the services he wants to use (operability is not evaluated 
at this time). For the purpose of measurement it is assumed: 

• that the service is available and not barred for any reason; 

• routing is defined correctly without errors; and 

• the target subscriber equipment is ready to answer the call. 

Further preconditions may apply when reasonable. 

The present document describes a set of use cases which are precisely defined to allow for comparability between 
different measurements, possibly performed by different parties. 
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Scope 



The present document specifies Test Profiles which are required to enable benchmarking of different GSM or 

3G networks both within and outside national boundaries. It is necessary to have these profiles so that when a specific 

set of tests are carried out then customers are comparing "like for like" performance. 



References 



The following documents contain provisions which, through reference in this text, constitute provisions of the present 
document. 

• References are either specific (identified by date of publication and/or edition number or version number) or 
non-specific. 

• For a specific reference, subsequent revisions do not apply. 

• For a non-specific reference, the latest version applies. 

Referenced documents which are not found to be publicly available in the expected location might be found at 
http://docbox.etsi.org/Reference . 

[1] ETSI TS 102 250-2: "Speech Processing, Transmission and Quality Aspects (STQ); QoS aspects 

for popular services in GSM and 3G networks; Part 2: Definition of Quality of Service parameters 
and their computation". 

[2] ETSI TS 124 008: "Digital cellular telecommunications system (Phase 2+); Universal Mobile 

Telecommunications System (UMTS); Mobile radio interface Layer 3 specification; Core network 
protocols; Stage 3 (3GPP TS 24.008)". 

[3] IETF RFC 348 1 . "TCP over Second (2.5G) and Third (3G) Generation Wireless Networks" . 

[4] IETF RFC 26 1 6: "Hypertext Transfer Protocol - HTTP/1 . 1 " . 



3 Definitions and abbreviations 

3.1 Definitions 

For the purposes of the present document, the following terms and definitions apply: 
session: continuous usage of a given service, e.g. a voice call or a data session 
NOTE: This may contain additional information. 

3.2 Abbreviations 

For the purposes of the present document, the following abbreviations apply: 

ACK Acknowledgement 

AMR Adaptive Multi-Rate 

BCP Best Current Practise 

DF Don't Fragment 

DNS Domain Name Server 

FQDN Fully Qualified Domain Name 

FTP File Transfer Protocol 

GR GPRS Register 

HLR Home Location Register 
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HTTP Hyper Text Transfer Protocol 

KPI Key Performance Indicator 

MOC Mobile Originated Call 

MTC Mobile Terminating Call 

PDP Pack Data Protocol 

PMTU Path Maximum Transmission Unit 

POP3 Post Office Protocol Version 3 

PSD Packet Switched Data 

QoS Quality of Service 

SACK Selective ACKnowledgement 

TCP Transmission Control Protocol 

TCP/IP Transmission Control Protocol/Internet Protocol 

VT Video Telephony 

WAP Wireless Application Protocol 



Measurement profiles 



Test profiles are required to enable benchmarking of different networks both within and outside national boundaries. It 
is necessary to have these profiles so that when a specific set of tests are carried out then customers are comparing 
"like for like" performance. 

It is recognised that many factors will affect comparability: 

• Number of sessions; 

• Sessions Duration; 

• Time between sessions; 

• Demanded QoS settings for data services; 

• Protocol settings (like TCP/IP settings for data services or AMR-settings for voice services); 

• Usage profile during the session; 

• Fixed network test equipment like test servers for data sessions; 

• User profile stored in the HLR or the GR; 

• Geographic location; 

• Type of location (indoor, hotspot, city, suburban, rural, train, etc.); 

• Speed when mobile; 

• Type of vehicle; 

• Type of antenna; 

• Handset type; 

• Handset hardware and firmware version; 

• Service being tested and limitations of service; 

• Network configuration; 

• Mobile users population density. 

For the points mentioned above where there is no recommendation or requirement in the present document, the settings 
experienced by a normal customer of the service under test in the network under test shall be used as a guideline. 
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As far as possible all particular values, e.g. timeout values, are named preserving the name of the respective KPI as 
defined in TS 102 250-2 [1]. 



4.1 



Classification of measurement environments 



For interpretation and comparability of test results it is important to know in which measurement environment the tests 
were performed. The environment classifications described below shall be used. Since the type of the measurement 
locations may be interpreted differently, the particular understanding of the location type determining a category shall 
be described in the results report. 

Stationary Tests 



Category 


Location Type 


Additional information 


S10: 


airports / railway stations / shopping centres and malls business districts and 
exhibition areas 


outdoor measurement 


S1I: 


airports / railway stations / shopping centres and malls business districts and 
exhibition areas 


indoor measurements 



Drive Tests / Walk Tests 



Category 


Location Type 


Additional information 


D1: 


Train Measurements 




D2: 


Urban Areas (medium cities) 




D3: 


Highways 




D4: 


Rural Areas (country roads) 




D5: 


Large cities 




W1: 


Walk Tests (indoor measurements) 




W2: 


Walk Tests (outdoor measurements) 




NOTE: Drive tests may be performed by incar using external antenna with an appropriate attenuation. 



4.2 Service profiles 

This clause describes recommended service profiles used for testing. 

4.2.1 Telephony 

For all Telephony services it has to be stated, if the results were generated using MOC, MTC or a mix of both. The 
results for both types should be reported separately and should not be mixed. 

The default call duration used for telephony measurements should be 120 seconds. 

To achieve comparable statistics when performing a benchmark, there should be no fixed pause between calls. Instead, 
a fixed call window is defined in which the call has to be performed. If the call fails or drops, the next call attempt shall 
only be made when the next call window arrives. 

The minimum pause interval between two call attempts should be 30 seconds to prevent network related problems 
between connection release and the next establishment (e.g. signalling in the PSD or mobility management). 

4.2.1.1 Voice telephony- 

Voice Telephony should be tested either in MOC or in MTC direction. The following call durations shall be used: 
CD1: 10 seconds for call setup testing; 

CD2: 120 seconds for typical tests, default call duration; 

CD3: 300 seconds for stability tests; 

Call Window: Call Duration + 30 seconds, (for the setup and release phases) + 30 seconds (for the minimum pause 
interval), for the default call duration this results in 180 seconds. 
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Timeout values: 

• Service Accessibility-Telephony Timeout: 20 seconds; 

• Setup Time-Telephony Timeout: 20 seconds. 



4.2.1 .2 Video Telephony 

Video Telephony should be tested either in MOC or in MTC direction. The following call durations shall be used: 



CD1 
CD2 
CD3 



10 seconds for call setup testing; 

120 seconds for typical tests, default call duration; 

300 seconds for stability tests; 



Call Window: Call Duration + 60 seconds, (for the setup and release phases) + 30 seconds (for minimum pause 
interval), for the default call duration this results in 210 seconds. 

NOTE: The reason for the longer call window for video telephony is that the call setup time for VT calls can be 
longer due to terminal capability negotiations. 

Timeout values: 

• VT Service Non- Accessibility Timeout: 20 seconds; 

• VT Service Access Time Timeout: 20 seconds; 

• VT Audio/Video Setup Time Timeout: 30 seconds. 



4.2.2 Messaging Services 



For all messaging services it is important that the recipient of a message is not interrupted by the next message while 
retrieving the previous one. For this reason it is important that the interval between sending two messages is larger than 
the 95 % percentile of the end-to-end duration, unless measures are taken to avoid this kind of interference. 

It should be noted, that mobility of either the sender or the receiver or both of a message can have an impact on the 
results. Therefore it is recommended that measurements are not only performed stationary, but also with mobility of one 
or both participants. In all cases the used scenario has to be stated. 

4.2.2.1 SMS 

SMS should be tested either in MO or in MTdirection with respect to the mobiles used as measurement probes. The 
SMS should be 120 characters long and use different characters to test content integrity. A reference SMS can be found 
in the appendix. 

The interval between two consecutive SMS shall be 70 seconds. 

The time window of measurements for calculating the Completion Rate SMS shall be 175 seconds. 

Timeout- values: 

• Service Accessibility SMS MO Timeout: 65 seconds; 

• Access Delay SMS MO Timeout: 65 seconds; 

• End-to-end Delivery Time SMS Timeout: 175 seconds. 

4.2.2.2 MMS 

MMS should be tested end to end. That means a MMS send by A-Party should be received by B-Party using also a 
mobile phone. The advantage of this testing is, that the MO direction at A-Party and the MT direction at B-Party can be 
measured. Both directions together are the end-to-end Parameters described in TS 102 250-2 [1]. 
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The following MMS sizes shall be used: 

• MMS1: 2KB; 

• MMS2: 28 KB; 

• MMS3: 90 KB. 

Reference MMS can be found in the appendix. 

If the MMS is not delivered at the destination mobile after the MMS End-to-end Failure Ratio Timeout, the MMS 
delivery is considered failed. MMS delivered after this time is not taken into account for end-to-end delay, but into 
end-to-end failure ratio. 

Timeouts for MMS over GPRS: 

The timeouts for MMS Send, Retrieval and End-to-end Failure are dependent on the MMS size. For GPRS all MMS 
uploads with less than 5 kbit/s and all MMS downloads with less than 10 kbit/s are considered to be cut-off. 

MMS send failure ratio (MO) Timeout & MMS Send Time (MO) Timeout: 

195 seconds + MMSsize[kByte] x 8/5 

MMS Retrieval Failure Ratio (MT) Timeout & MMS Retrieval Time (MT) Timeout: 

195 seconds + MMS size [kByte] x 8/10. 

The fixed part of 195 seconds incorporate the time for PDP context activation and WAP activation and shall be used as 
a whole, i.e. the single timeouts for PDP context and WAP activation shall not be considered. 

MMS end-to-end Timeout Timeout & MMS End-to-end Delivery Time Timeout: 

(590 + Size[kByte] x 8/5 + Size[kByte] x 8/10) [seconds]. 

The fixed part of 590 seconds incorporate the time for PDP context activations, WAP activations and notification and a 
security margin. It shall be regarded as a whole, i.e. the single timeouts shall not be considered. 

MMS Notification Failure Ratio Timeout & MMS Notification Time Timeout: 120 seconds. 

Timeouts for MMS over UMTS: 

The timeouts for MMS Send, Retrieval and End-to-end Failure are dependent on the MMS size; 

The respective required minimum upload and download data rate is for further study; 

MMS send failure ratio (MO) Timeout & MMS Send Time (MO) Timeout: for further study; 

MMS Retrieval Failure Ratio (MT) Timeout & MMS Retrieval Time (MT) Timeout: for further study; 

MMS end-to-end Timeout Timeout & MMS End-to-end Delivery Time Timeout: for further study; 

MMS Notification Failure Ratio Timeout & MMS Notification Time Timeout: 120 seconds. 

4.2.3 Data services 



4.2.3.1 Circuit switched 

Circuit switched data services shall be tested 100 % MOC. Call duration shall be either 300 seconds or is defined by the 
usage profile used during the data session. The pause interval between call attempts shall be 30 seconds. The usage 
profile used during the data session is defined in clause 4.3. 
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4.2.3.2 Packet switched 



Packet switched data services shall be tested 100 % MOC. Session duration shall be either 300 seconds or is defined by 
the usage profile used during the data session. The pause interval between session setup attempts shall be 30 seconds. 
The usage profile used during the data session is defined in clause 4.3. 

Timeout values: 

• Service Accessibility Timeout: 150 seconds + IP-Service Access Timeout. 

• Setup Time Timeout: 150 seconds + IP-Service Access Timeout. 

• IP-Service Access Timeout (& IP-Service Setup Time Timeout): 

FTP (UL&DL): 30 seconds; 

HTTP: 30 seconds; 

E-mail (POP3 & SMTP): 60 seconds. 

• Completed Session Timeout (& Session Time Timeout); 

• Data Transfer Cut-off Timeout: 

Over GPRS: 

UL: File size[kByte] x 8/5; 

DL: File size[kByte] X 8/10; 
Over UMTS: 

UL&DL: File size[kByte] x 8/50. 
Dual mode: The average between the timeout over GPRS and UMTS shall be considered. 

• Attach Timeout: 75 seconds. 

It might occur that the mobile station sends more than one attach request towards the SGSN, since retries are necessary. 
A maximum of four retries are possible (timer T3310 expires after 15 seconds for each attempt, see TS 124 008 [2]. 

• PDP Context Activation Timeout : 150 seconds. 

It might occur that the mobile station sends more than one PDP context activation request towards the SGSN, since 
retries are necessary. A maximum of four retries are possible (timer T3380 expires after 30 seconds for each attempt, 
see TS 124 008 [2]). 

4.2.3.3 Streaming 

For Further Study 

4.3 Usage Profiles for Data Sessions 

For all data measurements TCP settings may be chosen at will. 

If the measurements shall be used for comparison with other networks the following settings shall be used on the 
measurement client (based on the assumption, that the majority of the customers will use Microsoft WINDOWS XP 
Professional SP1 English): 

• Maximum Segment Size between 1 380 Bytes and 1 460 Bytes; 

• TCP RX Window Size = 16 384 Bytes; 

• SACK enabled; 
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ECN disabled; 

TCP Window Scaling disabled; 

TCP Timestamping disabled; 

PMTU Discovery disabled (but DF-bit set); 

TCP Fast Retransmit; 

TCP Fast Recovery enabled; 

Delayed ACK enabled (200 ms). 

NOTE 1 : The recommended TCP settings represent one of the possible (out-of-the-box) implementations of a 
common operating system of a client. The reason why this option was chosen was due to the fact that 
these are closer to the "default" user settings. 

These settings are not and should definitely not be considered as the preferred or optimised TCP settings over 2.5/3 G. 
A good reference for optimised TCP parameters over cellular networks like 2.5/3 G is the RFC 3481 [3] which is of the 
Best Current Practise (BCP) category in the IETF published on February of 2003. 

NOTE 2: Although the same TCP parameters may be used for benchmarking purposes, when different OSs are 
used in the tests, different results maybe obtained. This is due to the different implementations of the 
TCP/IP stack in the different OSs. This needs to be considered when comparing the results from a 
benchmarking exercise especially when the client and/or server OSs of the compared networks are not the 
same. This is a limitation which should not be overlooked, and a possible solution is to use the same 
version of the OSs in the client, as well as the server for all networks compared in a benchmark campaign 
(this does not imply that the OS of the client must be the same as the server). 

NOTE 3: Proxy servers installed in the networks IP core network may act as the TCP peer instead of the application 
server the tests are performed against. 

Measurements with other settings may not be called conforming to the present document. There is no 
preference concerning the used operating system as long as these settings are used. 

For all tests a dedicated test server should be used as a well defined reference. Under no circumstances 
should a commercial server (e.g. www.yahoo.com) be used, since the content on such a server may 
change over time. This makes later reproduction of the results impossible. 

The test server should be identified by an IP address and not by its FQDN in order to avoid issues with 
DNS lookup and including the DNS caching strategies of the used operating system into the 
measurement. 

The TCP settings of the server tested against should also be recorded. Since the number of host operating 
systems for internet servers is larger than on the client side, no detailed recommendation concerning the 
TCP settings of the server is given. However, the TCP stack of the reference server should at least be 
capable of the following: 

Maximum Segment Size between 1 380 Bytes and 1 460 Bytes; 

TCP RX Window Size > 4 096 Bytes; 

SACK enabled; 

TCP Fast Retransmit; 

TCP Fast Recovery enabled; 

Delayed ACK enabled (200 ms). 

The measurement of data services should take place against a reference server only used for testing purposes. The 
reference server should be connected to public the internet with a link capable of carrying the accumulated traffic of all 
mobiles testing against that server (e.g. if a benchmark with 4 networks is performed, the server should be able to 
deliver at least 4 times the maximum nominal speed of a single wireless link). There should be no bias concerning the 
IP connectivity to this server from a specific operator (e.g. bandwidth or hop-count). 
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The server application (e.g. web server, FTP-server, mail server) should behave similar to the majority of servers used 
on the internet. 

ETSI will provide a master reference server, from which reference web pages can be mirrored and which can also be 
used as a reference server. Please check availability and usage with ETSI Secretariat. 

For GPRS the Multislot Class of the measurement terminal shall be stated in the results report. 

4.3.1 Web browsing using HTTP 

For the measurement of data services the reference server should contain a static version of a reference webpage of 
fixed size that is designed similar to a common webpage. 

The browser used for testing should behave similar to the browser used by most of the customers. This is currently the 
Microsoft Internet Explorer. It should be able to support the same HTTP capabilities and headers and open the same 
number of parallel download threads to download the content as the reference browser. 

After one test cycle (one download of the reference page) the local cache of the browser should be cleared and it should 
be made sure, that all TCP connections between the server and the client are closed (no HTTP keep alive). There should 
be a pause of at least 6 seconds between the cycles. 

The webpage used for testing should contain a typical mixture of text and graphics. The number of objects and their size 
should also be typical. The website should not contain any dynamic content like Java or JavaScript and all URLs used 
should be conforming to RFC 2616 [4] (e.g. should not contain blanks or illegal characters). 

• For UMTS the following web page with 270 kb and 24 objects shall be used. 

Integrate the file(s) in the document (e.g. as ZIP file or link) 

For the test, only HTTP download should be used. HTTP upload shall not be used. 

Testing of content integrity is not mandatory for this test, but highly recommended. 

The appendix contains a description of a typical reference webpage of approximately 75 KB. 

4.3.2 E-Mail access 

E-Mail Access should take place against a reference mail server. 

A cycle should consist of mail upload using SMTP and mail download using POP3. Both upload and download should 
represent typical user behaviour. 

After a test cycle, all TCP connections to the server should be disconnected. 

Testing of content integrity is mandatory for this test. 

Currently, IMAP and other mail protocols are not taken into account. 



4.3.3 File Transfer using FTP 



FTP testing should take place against a reference FTP server. The server should support the standard FTP commands 
and both active and passive mode transfer of data. The server should allow only a single download thread to the client 
and not multiple connections from one client at the same time to speed up the download. There should be no bandwidth 
limitation on application level. 

4.3.4 File Sharing using UDP 

TBD 
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4.3.5 Synthetic tests 

4.3.5.1 UDP 

TBD - Reference to iPerf. 

4.3.5.2 ICMP 

TBD - Reference to TPING. 

4.3.5.3 TCP 

TBD - Reference to TPING. 
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Annex A (informative); 
Reference SMS 



For Further Study 
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Annex B (informative): 
Content integrity checking 



B.1 HTTP 

For Further Study 



B.2 FTP 

For Further Study 



B.3 MMS 

For Further Study 
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